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In 2002, the North Carolina Department of Public Instruction (DPI) initiated an innovative and 
ambitious program for improving student learning through the effective use of instructional 
technology. Until that point, most technology projects in schools were geared toward integrating 
technology into the curriculum, and they tended to focus on professional development for 
teachers and the acquisition of equipment. Evidence that technology integration was making a 
difference in student learning was limited. In the belief that technology can be a catalyst for 
improving student outcomes, and armed with a sound understanding of the research on school 
improvement, DPI developed the IMPACT model for schools. The model goes beyond 
traditional approaches to technology integration by encompassing all of the elements that 
research suggests are components of an effective school-based technology and media program: 
leadership, collaboration, professional development, a media coordinator, a technology 
facilitator, flexible scheduling, infrastructure, resources, evaluation, and an adequate budget. To 
pilot the model, DPI took the bold step of using their funding from the Enhancing Education 
Through Technology (EETT) component of No Child Left Behind (NCLB) to award IMPACT 
grants to eleven resource poor schools across the state. Each of the eleven grants was for three- 
years and 1.5 million dollars, which provided the IMPACT schools the time and resources they 
would need in order to fully implement the model. 

With such a heavy commitment of resources to individual schools, educators and policymakers 
in North Carolina - and across the nation - were interested in finding out whether the IMPACT 
model does indeed improve student learning. Thus, in 2003, DPI sought and was awarded a grant 
from the U.S. Department of Education (ED) to study the implementation and impact of the 
model. Looking at North Carolina Educational Technology (LANCET) was one of ten research 
projects supported by the Evaluating State Educational Technology Programs (ESETP) initiative. 
DPI’s partners in the LANCET study were the Friday Institute for Educational Innovation at 
North Carolina State University (NCSU) and the SERVE Center at the University of North 
Carolina at Greensboro (SERVE). Researchers at NCSU conducted experimental and quasi- 
experimental research on several aspects of the IMPACT model, including the effect of the 
model on student achievement. SERVE’s role in LANCET was to help the IMPACT schools 
develop the internal capacity for conducting formative evaluation of their individual projects, 
focusing on uses of data to make decisions for improving the implementation of their projects. 
The purpose of this document is to share insights into and lessons learned from SERVE’s work 
with the IMPACT schools as they implemented and evaluated their projects. 

To help the schools learn how to plan and conduct the evaluation of their IMPACT projects, 
SERVE provided professional development, technical assistance, and resources to school teams. 
To support the work, SERVE developed the Capacity for Applying Project Evaluation (CAPE) 
suite of resources and tools, which includes an evaluation framework, a compendium of 
evaluation instruments, and a professional development model geared toward capacity building 
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(http://www.serve.org/Evaluation/Capacity/) . With additional funding from the Microsoft 
Corporation’s Partners in Learning (PIL) program, SERVE was able to expand the CAPE model 
and services to educators in other states and to study how IMPACT school administrators’ and 
teachers’ evaluation skills and understandings changed, how individuals and organizations 
responded to the evaluation capacity-building effort, and what factors seemed to influence the 
schools’ capacity for planning and conducting formative evaluation. 


Conceptual Framework 

In developing CAPE, SERVE delved into the research literature on school change, the adoption 
of educational innovations, and capacity building. The following table illustrates how changes in 
schools are influenced by three driving forces: engaging moral purpose, understanding the 
change process, and building capacity. Capacity building is in turn dependent on the extent to 
which educators’ knowledge, skills, and attitudes are changed, resources are available and 
utilized, educators engage in a professional community around the innovation, the program is 
coherent, and leadership is shared among key players. Note that the professional literature 
addresses innovations in general; for CAPE, the innovation is the capacity for project evaluation. 


Foundation 
Drivers for 
Change in 

1. Engaging Moral Purpose 

Engaging teachers’ beliefs; the need or motivation to adopt the innovation 
(Fullan, 2005). 

Schools 

The desired 
change is 
defined as the 
adoption of 
an innovation. 

2. Understanding the Change Process 

Engendering ownership of changes necessary to adopt the innovation 
(Fullan, 2005; Hall & Hord, 1984; Horsley & Loucks-Horsley, 1998; 
Rogers, 1995; Marzano, R., Waters, T., & McNulty, B. (2005). 

3. Building 
Capacity 

Collective and 

A. Knowledge, Skills, and Attitudes of 
Individuals 

(Guskey, 1986, 2000) 

(Fullan, 2005) 

ongoing policies, 
strategies, resources, 
and other actions to 

B. Resources - infrastructure, tools, people, 
money, time necessary to adopt the innovation. 


tlllvi V/ til Vl CVV/L-AV/XXO Lv/ 

increase 
organizational 
power to adopt the 

Shared 

Identity 

Motivation to 

C. Professional Community 

(Wenger, McDermott, & Snyder, 
2002) 


innovation. 
(Newmann, King, & 
Young, 2000, as 
cited in Fullan, 

2005, p. 40) 

work together 
to further 
adoption of the 
Innovation 

D. Program Coherence 

(Newmann, Smith, Allensworth, 
& Bryk, 2001) 


-Lllllvy V 

(Fullan, 2005) 

E. Shared Leadership 

(Lambert, 1998, 2002) 


CAPE Conceptual Framework 
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The elements of this conceptual framework provide a basis for thinking about building capacity 
for project evaluation and about promoting changes in practice. Fullan (2005) explains the 
engagement of “moral purpose” as the activation of a) educators’ fundamental beliefs about what 
is valuable for children and b) educators’ ownership of the desired change. In the case of many 
educational technology projects, this means that an important factor in a project’s ultimate 
success may hinge on whether a) teachers believe that the innovation will benefit their students 
and b) the extent to which teachers see themselves playing a role in realizing those benefits. It is 
equally important that the people involved in a project understand that they are going through a 
change process; the better prepared they are for the change, and the better they understand the 
process they are going through, the less likely they may be to resist the change. 

The framework expands the concept of capacity building beyond traditional notions of increasing 
knowledge, acquiring skills, and changing attitudes to promoting and supporting the actual use of 
the knowledge, skills, and attitudes, along with the use of necessary resources. A final crucial 
aspect of capacity building is shared identity or collaboration, in which parents, students, 
teachers, and administrators are mutual learners and leaders, and in which everyone has a shared 
vision of how the project works in order to achieve the desired outcomes. 


Lessons Learned 


In order to understand educators’ perceptions of their experiences in developing the capacity for 
project evaluation, SERVE implemented a series of site visits and conducted focus groups and 
individual interviews with key participants in the IMPACT projects. The observations and 
lessons learned presented here derive primarily from a content analysis of the qualitative data 
from the focus groups and interviews. Focus group participants and interviewees included 
project management team members (school administrators, technology facilitators, media 
coordinators, classroom teachers, and in some cases, a district technology staff person) as well as 
members of the IMPACT schools’ staff who participated in, but were not directly involved in, 
managing their IMPACT project. 

Lessons are overarching insights into what might be learned about capacity building for 
evaluation in the IMPACT schools. Suggestions, which go beyond the data, are an attempt to 
translate the lessons and insights into practical steps that school and district educators might find 
useful in their own projects. Observations capture Ah ha! moments from SERVE’s professional 
development and technical assistance work with EETT schools. Recommendations are offered 
to funding agencies and policymakers to help them effectively plan evaluations at a program 
level. 


Lesson 1: Different stakeholder groups see different purposes for the evaluation of 
educational programs and projects. 

At the federal level, policymakers tend to see evaluation as a means for gathering evidence that a 
program is meeting its goals and that the money is being spent as Congress intended. At the state 
level, program leaders are often under pressure to provide data to the federal, and sometimes 
state, government to show that the priorities are being addressed and program regulations are 


Lessons Learned about Evaluation Capacity Building, November, 2007. 
SERVE Center at UNC-G 


3 



being followed. At the same time, state-level program leaders want data they can use to ascertain 
how well projects are being implemented in local education agencies, which project processes 
and strategies are the most effective, and how the projects are perceived by administrators, 
teachers, and the community. University professors and other researchers who conduct external 
evaluations are interested in contributing to the professional knowledge base and literature by 
investigating how particular educational strategies impact student outcomes. All of these varying 
purposes for evaluation are fine and good, but school-based educators do not see much relevance 
for themselves and their schools. What matters most to teachers and principals is getting the most 
out of the evaluation in order to benefit their students. Teachers, in particular, are generally less 
than enthusiastic about doing evaluation for the sake of accountability or research, but if they 
believe that a particular innovation or strategy will benefit their students, they are more likely to 
be willing to actively participate in its evaluation. 

According to many textbooks, the purpose of an evaluation is formative, i.e., intended to help 
managers monitor and adjust their project in order to improve its effectiveness, or summative, 
providing data that funding agencies use for making decisions about whether to continue 
financial support. Schools should collect and use data formatively to ensure the successful 
implementation of their projects; state education agencies need summative data they can use to 
decide whether to continue funding the projects; and the federal government needs summative 
data they can show to Congress as evidence that the goals of the federal program are being 
addressed. Problems can arise when the needs and interests of the stakeholder groups are in 
conflict. For example, a funding agency might require specific kinds of data to be collected by a 
particular instrument. To ensure that they get the data they need, the funding agency might 
require that the project evaluation plan they have approved be implemented as planned. In the 
real world of schools, however, educators who are collecting data to use in making programmatic 
decisions need some measure of flexibility in collecting data and implementing the plan. 

At the project level, teachers and administrators also need to understand why they are collecting 
data, what project leaders expect to learn, and what is going to be done as a result. Ultimately, if 
you want to build capacity for evaluation in schools, the most valuable (or moral ) purpose for 
evaluation is to arm decision-makers with the data they need so they can get the most out of their 
project in order to benefit students. 

Suggestion - For educators in schools, learning how to use evaluation data is more 
important than learning how to develop and implement plans. When planning 
professional development aimed at helping schools develop their capacity for evaluation, 
don’t base the activities on helping participants learn how to evaluate their projects. 
Rather, approach the professional development as a means for helping project teams get 
the most out of their projects. In the end, it isn’t about evaluation; it’s about doing what’s 
best for students. 

Informants, i.e., those who are asked to provide data, may be more likely to spend valuable time 
completing surveys, participating in interviews or focus groups, or providing samples of their 
work if they see the value in doing so. One educator said: “We were . . . told what to do but 
really, it would have been better to have an understanding of why we were doing what we were 
doing or. . .given any suggestions of what we could learn from this.” Since the success of any 
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project evaluation rests on having access to the necessary data, the first and best step to assure 
that project participants actively support evaluation efforts is to make sure they understand that 
their efforts will ultimately benefit students. 

Suggestion - Provide feedback to stakeholders based on data collected. One time- 
honored way to encourage project participants and stakeholders to provide data is to 
provide them with summary reports and feedback based on the data they provided. 
People want to know that they are contributing — that the information they provide 
actually matters. Often, they want to know how their responses compare to those of the 
larger group. This strategy also allows those providing data to use that information for 
their own formative purposes. A classroom teacher’s explanation illustrates this point: “I 
think sometimes if we can know the results of the evaluation it would help us. . .[to] can 
change things or make suggestions as to how things could be improved or whatever. . .if 
we know other people’s opinions.” 


Lesson 2: Language matters. 

As a profession, education is notorious for its jargon. Not only do we create terms and acronyms 
that are seldom used by the general public (e.g., AYP, IEP, EETT), but also use terms differently 
among ourselves. Consider, for example, the term evaluation. For educational program leaders, 
an evaluation is generally accepted to be an assessment of the quality of a program or 
intervention, based on rigorous and systematic collection and examination of appropriate data. 
Classroom teachers and school administrators, however, often associate evaluation with 
“grading” — assigning values to reward good work or punish poor performance. If schools are to 
build their capacity for evaluating educational programs and projects, it is crucial that teachers 
and administrators understand that evaluation of educational projects is about collecting and 
analyzing data in order to make good decisions about what is or is not working, rather than 
making judgments about whether the school, teachers, and students are good or bad. 

Consider also the practical aspects of the language of evaluation. One of the best approaches to 
building capacity for evaluation is for teams of educators to work together. But what happens 
when individual team members use terms differently? For example, the term “benchmark” can 
have a variety of meanings. If an administrator who has studied evaluation in graduate school is 
asked to define the term “benchmarks,” she might say something about specific levels of quality 
used as targets or points of comparison for data. If a classroom teacher or principal were asked to 
define benchmarks, they might think of a product called Benchmarks, which is a published series 
of student assessments. Conversations among educators with these very different points of 
reference can be confusing and frustrating. The lesson learned from the IMPACT schools is that 
it is less important that everyone involved in an evaluation have the “right” definition of any 
particular term (e.g., goal, objective, standard, or indicator) than it is that they all use the same 
definition. 

Suggestion - Early in the project, establish and maintain a “team glossary, ” created by 
the group responsible for managing the project and its evaluation. As terms come up in 
planning conversations, note them on poster paper, white board, or an electronic file, and 
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take the time to discuss, select, and record a shared definition of each. Keep the glossary 
current and available for reference by everyone working on the evaluation. 


Lesson 3: It’s important for all of the players in an evaluation to understand each other’s 
roles. 

In the case of large statewide programs such as the IMPACT initiative in North Carolina, 
individual schools could be participating in more than one evaluation. Like the other ESETP 
projects, the IMPACT schools participated in multi-faceted experimental and quasi-experimental 
research efforts as well as the formative and summative evaluations of their projects. In addition 
to the NCSU researchers and external evaluators, the schools were visited by DPI consultants 
whose role was to help the schools implement the projects, and SERVE staff whose aim was to 
help the project teams with formative project evaluation. Frequently, the teachers did not 
understand who the individuals were, the organization they represented, or their purpose for 
being in the school. All they knew was that they were being asked to provide data, which meant 
extra time and effort on their parts. Given the complex nature of the statewide initiatives, and 
given the fact that schools often experience high turnover rates in personnel, it’s crucial to clarify 
for everyone - especially the principals and teachers - the roles of the respective organizations; 
the roles of, and expectations for, everyone involved; who is collecting what data; and what will 
happen with the data. If at all possible, this information should be provided as early as possible, 
and there should be updates as the program evolves and as new personnel come on board. 

Suggestion - Clarify the roles and responsibilities of all individuals participating in 
evaluation activities at all levels, and have a plan for making sure that they are 
understood by all involved. An external evaluator hired by a funding agency and charged 
with ascertaining project accountability will be received by a school principal differently 
than one voluntarily contracted by that same administrator to help her school replace 
intuitive decision-making with evidence-based planning. In the same vein, a university 
professor contracted to observe specific classroom teaching strategies in order to study 
their effect on student achievement will have a very different relationship with teachers 
than will a professional development coach who works with the teachers on the use of 
student data for improving instruction. 


Lesson 4: The time that teachers have for working on a project evaluation is limited. 

Principals and teachers often question whether the benefits of planning their project evaluations, 
collecting data, analyzing data, and reporting the results are worth the time and energy required, 
especially when they have a plethora of other priorities, such as meeting the requirements of No 
Child Left Behind. Classroom teachers are especially concerned about how time spent on the 
evaluation might detract from working directly with their students. Principals, in turn, are 
concerned about teachers being overwhelmed with the “additional” responsibilities. 

If project evaluations are conducted completely by project staff — defining an internal evaluation, 
in contrast to an external evaluation managed by outsiders — the question becomes, “Who is 
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responsible for data collection and other activities?” There may be a tendency for these 
responsibilities to be pressed onto teachers, since they are either the source of useful data or have 
access to the sources, such as students, student activities or products, or parents. This becomes a 
very real management challenge for administrators, who are leery of imposing on already-busy 
educators. 

Suggestion: - If steps have already been taken to assure that teachers and administrators 
recognize the value of the evaluation effort in which they are being asked to participate, 
their actual participation might not be perceived as an imposition. If educators do feel 
imposed upon, however, it is a good idea to use time that has already been dedicated for 
planning and decision making activities, such as school staff meetings, for completing 
surveys, reflective journals, activity checklists, or other data-collection instruments. As 
one of the IMPACT teachers explained, it is a good way to “recognize the demand for 
time imposed by evaluation activities and respect participants’ need for time.” 


Lesson 5: Developing the capacity for formative evaluation can change not only projects, 
but also the people implementing the projects. 

In the interviews and focus groups conducted by SERVE, most of the IMPACT schools reported 
making changes in their project implementation as a result of the findings of their evaluation. 
They also described changes in their evaluation skills and in understandings, attitudes, and 
behaviors related to evaluation. Many expressed an increased understanding of aspects of 
evaluation processes. Interviewees and focus group participants indicated they had a better 
understanding of differences in, and purposes for, summative and formative evaluation. Many 
also felt as though they had gained understandings of the various data-collection tools that might 
be used to evaluate technology projects in schools. Staff members reported that they will likely 
use at least some of what they learned about formative evaluation in the future, to help them 
make adjustments to their projects. 

Interview and focus group data revealed that the collection of data examining the quality of their 
project implementation made teachers more attentive to particular aspects of their instructional 
practices, particularly if the data collection methods included a reflection component, such as 
journals or activity logs. These changes in teacher focus seem to be a result of evaluation 
activities heightening teachers’ awareness of particular practices, making them more conscious 
of their performance and more likely to adapt it to expected outcome standards. One teacher 
interviewed explained that, because she was asked in a survey about how often she used a 
particular strategy, she asked herself, “okay — how can I incorporate that into my lessons more?” 

Influences were also seen in terms of student assessment or in teachers’ use of data to inform 
their own instruction. For example, one teacher learned how to design and use rubrics as part of 
her school’s project evaluation, then used her new skills to assess student performance. As an 
educator said, “You do a unit and you ask yourself, you either say - I’ll never do that again or 
that was really great but I need to do this part different. . . I think this has taught me to write it 
down, make a record. . .which I don’t know if I did before.” 
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Suggestion: Organizational change is highly dependent on change in individuals, and 
change in individuals can be, but does not have to be, difficult. It helps tremendously if 
those who are involved understand the nature and scope of the changes in both the 
program and (potentially) themselves. In retrospect, the IMPACT teams would have been 
well served if they had had a framework for monitoring, thinking about, and discussing 
the changes they and their schools were going through, such as the Concerns Based 
Adoption Model (Hall & Hord, 1984) or Fullan’s (2001) work related to leadership and 
change. 


Lesson 6: In developing the capacity for formative evaluation, teachers and administrators 
do not need to become evaluation experts, but they do need to develop specific evaluation 
skills. 

Although school-based project evaluators do not need to become experts in evaluation designs or 
methodologies, it is helpful if they learn to focus the scope of their data collection efforts on 
answering the key evaluation questions. Particularly with increased emphasis on measurable 
student achievement and the advent of current technologies, educators tend to collect too much 
data in the course of doing an evaluation, rather than not enough data. In the focus groups and 
interviews, when educators were asked what they would do differently if they had their 
evaluation to do over again, many recommended simplifying their data collection: “I think we 
would narrow our focus on the kinds of things that we need to look at. . .And I don’t think that I 
would try to collect everything all at once.” A good mantra for data collection is “collect all of 
the data you need and only the data you need” in order to answer the evaluation questions. 

To enhance the rigor of an evaluation, much of the professional literature on evaluation methods 
encourages the use of multiple measures for collecting data about a particular subject of inquiry, 
which is referred to as the triangulation of data. However, for the purposes of a school-based 
formative project evaluation, project leaders should consider whether that increased rigor 
outweighs the costs of collecting, organizing, and analyzing additional data. Our experiences 
with schools tell us that sometimes it is not worth the extra effort. School staff can often find out 
what they need to know from a single, reliable data source. We have also noted that it is possible 
to “go to the well” too often, upsetting project stakeholders by asking them to provide the same 
data for multiple evaluation efforts. 

Suggestions - Rather than adding a lot of data collection procedures, try making simple 
adjustments to data already being collected. It may be useful to adjust existing data 
collection processes (e.g., surveys, professional development questionnaires, student 
assessment, resource management records) to maximize their usefulness for the defined 
evaluation purpose. Add a few items to an existing school staff survey rather than 
creating an entirely new one, or modify an established rubric used to assess student 
products to address specific desired outcomes. For example, when faced with the 
challenge of assessing teachers’ use of specific technology in their teaching, one school 
project team looked to lesson plans as evidence, and discovered that by making a few 
simple adjustments to their existing lesson plan template, they could easily use it as a data 
source for assessing teacher technology use. 
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Also, take a look at the data from integrated learning systems (ILS ) — software programs 
that adapt instruction or practice to individual students based on their performance. These 
systems collect student performance data that can be configured to report the results at 
the student, class, or school level. If teachers and administrators develop the skills and 
understandings necessary to analyze, interpret, and apply this wealth of information, ILS 
data can be effectively used not only to modify instruction, but also to support a variety 
of evaluation purposes. 

Teachers and other staff members involved in the evaluation need training and support that will 
help them build the knowledge and skills that will enable them to successfully identify 
appropriate instruments, complete surveys, collect and organize necessary data, analyze that 
data, and then interpret meaning from the analysis. In large programs such as the IMPACT 
program, where there are a variety of organizations that can offer support (DPI, SERVE, NCSU), 
the project teams need to know who can help them and how to engage their assistance. 

“Teach us how to collect data that will support that what we are doing is actually bringing 
about our goals,” one educator said. Another recommended providing professional 
development that addresses the following questions “How do you analyze data? If you 
collect data that says our lab is used by 150 students in the course of a week, what does 
that tell you? What should you look for? If lab use is what you’re studying, what should 
you look for...?” 

In some respects, teachers are already attuned to the notion of collecting and analyzing data; they 
just don’t think of it as evaluation. We often heard educators use the expression monitor and 
adjust, i.e., monitor the effectiveness of instruction, and if a particular strategy isn’t working, 
adjust it so that students can learn better. This is actually formative evaluation — but implemented 
informally, in real time, on a smaller scale than is typical in a project evaluation. Many teachers 
who participate in project evaluation will come to understand this: “The whole evaluation 
process is so commonsense, but not something I had ever done before. I think I did it but not 
necessarily formally.” Project leaders can help teachers and administrators who have not already 
adopted this way of thinking by showing them comparisons between project evaluation and the 
work that educators do in schools every day. 

By talking with others, educators usually have ideas about “how the project is doing,” but these 
conversations are a type of “informal data.” Oftentimes, the only difference between informal 
and formal data is that the latter is recorded for future reference and analysis, whereas the former 
exists only in memory, which runs the risk of loss or misinterpretation. Some of the IMPACT 
schools adopted strategies for tracking and organizing informal data, such as keeping a log of 
informal hallway conversations or developing rubrics or checklists based on standards to assess 
quality and document performance. 

At some point, the key question becomes, “What do the data mean?” Learning how to collect 
data isn’t all that different from what teachers normally do on a daily basis. Learning how to 
analyze and interpret the data is a whole new ball game. One of the many benefits of having a 
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project management team is that the individual team members bring different perspectives and 
insights to discussions about what the data mean and what should be done as a result. 


Lesson 7: Success in implementing a project evaluation — and likely of the project itself — 
depends in large measure on participants sharing a sense of identity around both the 
project and the evaluation. 

The IMPACT schools that were the most successful in implementing and communicating project 
evaluation processes and results generally seemed to be those where project participants and 
stakeholders had a good understanding of, and active involvement in, the evaluation. By 
involving project participants and stakeholders in the initial planning of both the project and the 
evaluation, teams can improve the quality of the evaluation and increase buy-in for activities that 
require the commitment of time and other resources. If it is not feasible to include all key 
stakeholders during the initial process, the next best option is to communicate with them soon 
after the planning and before the implementation. As one evaluation team member reported, 

“I think it helped us not to just look within our school but it was an extension to look into 
our district and our community to make sure that all stakeholders were involved, not just 
the teachers and the students, but make sure the parents were also involved.” 

Since formative evaluation is about helping project leaders monitor and adjust their project in 
order to maximize its effectiveness, an important first step in developing a viable evaluation is to 
ensure that all stakeholders involved have a good understanding of the project. It is important to 
bear in mind, however, that when evaluation is being conducted for formative purposes, the 
people involved in the project and its evaluation are likely to have a vested interest in the results. 
Data collection can be an emotionally challenging activity as project participants are asked to 
think and report critically on their — and others’ — activities, perceptions, and performance. Egos 
may be threatened if they are associated with specific project strategies, or issues of power (e.g., 
control of resources) may arise. Project leaders can play an important role in minimizing some of 
these challenges by creating an environment in which individuals associated with a project and 
its evaluation feel as though they are “in it together” — that they are members of one community 
and share the same understandings about project strategies and expected outcomes. 

One way in which IMPACT schools were able to clarify their project activities and desired 
outcomes, and to build consensus among project stakeholders was by developing a logic map — a 
graphical representation that illustrates how their project works. As one respondent noted, “I 
liked the logic map because it was a visual thing and you could see how it all goes together.” 

Through the logic mapping process, individuals involved in implementing the project evaluation 
are able to come to a shared understanding of what successful, quality implementation of project 
strategies looks like. Similarly, through this process, project participants and evaluators can 
reach common understandings of what outcomes ( objectives or goals ) are expected — or hoped 
for — as a result of implementing those defined strategies. More subtle, but of equal significance, 
is that all those involved have a shared understanding of how resources and strategies are 
expected to translate into the desired outcomes. Evaluators describe this as project logic — the 
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sum of various semantic, cause-effect or if-then statements that link project activities and results. 
When participants and stakeholders take the time early in the process to develop a shared 
understanding of what the project is going to do and what it is going to accomplish, the potential 
for misunderstandings and frustration can be greatly reduced. 

Suggestion - If logic-mapping strategies are thought to be valuable, apply them early in 
the project planning process and with enough flexibility that planning teams can 
illustrate their collective understanding of how their project works. It is not enough to 
simply require or provide preferences for a logic map (e.g., as part of a project proposal 
or program-level evaluation guidance). Logic maps are valuable primarily for the process 
of developing them, not for their mere existence. Their value is greatest if they influence 
project planning, implementation, and evaluation — as opposed to just evaluation — so 
project teams should map logic at the earliest possible point in project development. 

Logic maps should be revisited regularly, updated as the project and its evaluation evolve 
over time, and shared with all project stakeholders. 

From the IMPACT school data, we learned that while it may be tempting to provide templates or 
required formats for logic maps in an effort to simplify the process, this approach may in fact 
overtax or get in a team’s way, preventing them from developing a logic map that reflects their 
personal ideas of how strategies connect with objectives. While it might be helpful for teams to 
see a variety of logic mapping formats in order to get an idea of what they look like, imposing a 
mandate that they develop a logic map using a particular format might result in participants 
viewing the map as something to be completed — an artifact — rather than as the living planning 
document it should be. The important thing is that the logic represented in the map be sound and 
accurately reflect what the project is going to do and what its outcomes will be. 

Another successful strategy some of the IMPACT schools utilized in implementing their project 
evaluation was to share tasks among team members. As often occurs, when provided with the 
opportunity to share leadership responsibilities, educators who were not in positions that are 
traditionally associated with leadership, such as teachers or technology facilitators, often 
assumed leadership roles in their project implementation and evaluation. Differences in 
perspective often lead to evaluation questions that might have otherwise been overlooked, offer a 
broader base of experiences on which for interpreting the evaluation results, and help tailor 
communications to the various needs of audiences interested in evaluation results. One 
evaluation team member commented, 

“The process, too, of having the teachers in there with us instead of it just being the tech 
facilitator, the media specialist, and the principal, who have those bigger administrative 
roles as opposed to those teachers that are doing the day to day getting to the grind of 
things, they helped keep us grounded.” 

Suggestion - Develop a plan to actively share project and evaluation plans, activities, 
and results with stakeholders. Even when the majority of the evaluation activities are 
being led by a small team, communicating regularly with participant and stakeholder 
groups about the evaluation — including important questions, implementation progress, 
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data findings, and decisions — is likely to increase school-wide understanding and support 
of the project and its evaluation. 


Lesson 8: Communication is the glue that holds a project evaluation together. 

Communication is crucial to all aspects of successful project evaluation, whether it be for 
building a sense of community and buy-in, engaging participation in the evaluation, or sharing 
the results. The IMPACT schools used a number of strategies for communicating with staff 
throughout the project and evaluation, including whole schools meetings, dedicated time at 
weekly staff meetings, MTAC committee members and Team Leaders serving as liaisons 
between teachers and evaluation leaders, informal hallway conversations, and email. Whatever 
set of communications strategies were put in place, schools that communicated intentionally, 
frequently, and consistently about the project evaluation generally felt that they had a better 
understanding of their project. One educator illustrates this: 

“We lead the way by making sure that information is communicated to all areas. . .then 
the IMPACT committee, as we share that information with them, they, in return, take that 
information back to their grade level, to their respective grade levels, so we make sure 
that the line of communication is shared throughout the school so it doesn’t just stop with 
the IMPACT core team or the IMPACT team, or even the grade levels, because the 
teachers also, in turn, share with their teacher assistants. So, there’s sort of a line of 
communication that goes threaded throughout the whole process as far as information 
goes.” 

Suggestion - Early in the project, set aside time for the project team and school staff to 
discuss the best ways of communicating and to ensure that communication is meaningful 
and timely. Also, check to be sure that all of the stakeholders in the evaluation can and 
will participate in the designated lines of communication, e.g., that they will use email or 
digital discussion forums. From time to time, check to see how well communications are 
working. Are teachers getting the information they need? How about other stakeholders? 

The IMPACT schools felt more prepared to cope with evaluation tasks when they were well- 
informed in a timely manner about their roles and responsibilities and when provided with 
enough knowledge and skills to complete their expected tasks. Thus, it is important to plan 
communication not only within the schools but also among the schools and between the schools 
and program managers within the funding agency. 


Lesson 9: One size evaluation does not fit all. 

When we began working with the IMPACT schools, we thought we would be doing the schools 
a service by helping them learn how to plan and conduct formative evaluation of their 
technology projects and in so doing would help them develop capacity for evaluating any 
educational project. An underlying assumption was that most schools, especially those with 
limited resources, have little or no capacity for formative evaluation, i.e., formative evaluation in 
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the way that researchers and evaluators think about it. It turned out that most of the schools were 
doing evaluation in a variety of ways: collecting data for monitoring and adjusting classroom 
instruction, for accountability, for grant-funded projects, or in conjunction with the School 
Improvement Plan. In retrospect, we realize that if we had taken more time at the beginning of 
the process to fully understand each schools’ beginning level of capacity, the nature and scope of 
educators’ expertise, their school’s culture and operating style, and their specific needs, we could 
have built a measure of flexibility into the evaluation model and tailored the professional 
development and technical assistance in ways that would allow schools to more easily adopt or 
adapt the evaluation for their particular situations - what we have come to think of as 
developmentally appropriate evaluation support. Recognizing the evaluation work they are 
already doing conveys respect for the school, for the individuals involved, their knowledge and 
experience, and on-going efforts to improve their school projects for the benefit of their teachers 
and students. 

Another important aspect of the “One size doesn’t fit all” lesson is that when schools are 
adopting an evaluation model or evaluation instruments, there’s a good chance that they make 
modifications. For example, they might combine steps in a multi-step process, develop their 
own instruments, or change items on an approved instrument. On one hand, these modifications 
can increase ownership of, and participation in, the evaluation by teachers and administrators. 

On the other hand, the modifications can play havoc with fidelity of implementation. 

Suggestion: Before asking a school to adopt a new model or new instruments, take time 
to find out what is working, what constraints they might face, and what evaluation efforts 
they are able to take on; then provide evaluation support to meet their needs. 


Observations 


In working with EETT grant recipients in Mississippi as well as North Carolina, SERVE staff 
had a couple of important realizations. The first is that building capacity for project evaluation 
requires a lot more time, energy, and resources than traditional methods of professional 
development and technical assistance provide. Academies, institutes, workshops, and meetings, 
are fine, but what often makes the difference between great, moderate, or limited success is on- 
going guidance from someone who has expertise in evaluation and who understands and 
appreciates the cultures and exigencies of schools. It helps if the guidance is provided on-site as 
well as electronically. 

A second major observation is that if you want to help schools and districts develop and sustain 
the capacity for planning and implementing good project evaluations, always keep in mind that 
the focus of the effort should be on them and their projects, not on you, and not on learning how 
to do evaluation. Learning how to evaluate a project is not high on many teachers’ and 
administrators’ priorities; however, learning how to get the most out of finite resources so they 
can derive the most benefit for their students is a very high priority. In the end, it is not about 
project evaluation per se; it is about supporting the underlying purpose of the evaluation - 
educators having solid information and tools they can use in making sound decisions for 
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improving their projects, which translates into more effective teaching and improved outcomes 
for students. 


Recommendations 


Fullan (2005) suggests that in order for capacity building to be sustainable, it needs to be tri- 
level, i.e., to take place at the school, district, and state/federal levels. The lessons learned from 
the IMPACT schools suggest that there are certain things that policymakers and administrators at 
the state and federal and state levels can do that will ultimately improve their capacity for 
planning, conducting, and communicating the results of project evaluations. 

Program: Setting the Evaluation in a Context. Perhaps the most important thing that federal 
and state program planners can do for grantees is to make every effort to ensure that the grantees 
understand the purpose of the overarching program (beginning with the federal statute and the 
U.S. Department of Education’s priorities), see how their particular project fits into the overall 
program, and appreciate the importance of their formative and summative project evaluations. It 
would save the would-be evaluation team a great deal of time, frustration, and angst if evaluation 
were built into the overall program design and if all of the varying purposes for the evaluation 
were identified and addressed at the outset, i.e., as the program is being designed. It would also 
facilitate matters if the program design were to include (a) the purpose(s) of the evaluation; (b) 
expectations for the scope and nature of the evaluation, i.e., formative and/or summative, internal 
and/or external; (c) components of the project evaluations, such as evaluation questions, data 
sources, methods for data analysis, timeline; (d) roles of various stakeholders in planning and 
conducting the evaluation; (e) expectations for reporting the results; and (f) anticipated uses of 
the results. 

Expectations. All too often, funding agencies are disappointed with the quality of the evaluation 
plans in grant proposals and the quality of evaluation reports. This problem can be ameliorated 
somewhat if the Request for Proposals (RFP) or Grant Application Package outlines the agency’s 
expectations for the evaluation and provide enough structure such that inexperienced evaluators 
can develop a sound plan, but allow flexibility for the applicants to tailor the evaluation to their 
particular projects. A lot of time and energy could be saved if the RFP referred to specific data 
collection instruments that grantees can use for their project evaluations - so they won’t spend 
time developing their own instruments. 

Communication. Evaluation guidance or requirements are commonly passed downward from 
federal program officers, through SEA administrators (often multiple levels of them), to district 
administrators, and ultimately to school staff members. Project goals and objectives, 
management expectations, accountability requirements, and expectations of dissemination must 
be effectively communicated through all these levels. Similarly, evaluation data and findings are 
typically reported upward through those same levels and must be communicated consistently and 
in ways that meet the needs of each successive level of management. Communication among all 
these various levels must take place in ways that are timely — and timed — to effectively engage 
stakeholders in the evaluation and thus ensure the effective implementation of the evaluation. 
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Developing a plan for communicating with all stakeholder groups before project implementation 
begins and that encompasses new knowledge, roles and responsibilities, pathways and preferred 
methods for communication will go a long way in reducing confusion and unmet expectations, 
and toward gaining buy-in, understanding, and engagement. 


Lessons Learned about Evaluation Capacity Building, November, 2007. 
SERVE Center at UNC-G 


15 



References 


Fullan, M. (2001). Leading in a culture of change. San Francisco: Jossey-Bass. 

Fullan, M. (2005, August 30-31, 2005). Partners in Learning: Learning to lead change elite 
course, Redmond, WA. 

Guskey, T. R. (1986). Staff development and the process of teacher change. Educational 
Researcher, 15(5), 5-11. 

Guskey, T. R. (2000). Evaluating staff development. Thousand Oaks, CA: Corwin Press. 

Hall, G. E., & Hord, S. M. (1984). Change in schools: Facilitating the process. Albany, NY: 
State University of New York Press. 

Hergert, L.F., Mundry, S.E., Kolb, F.A., Rose, R., & Corro, J. (1989). Making change. Boston: 
The Network. 

Horsley, D. L., & Loucks-Horsley, S. (1998). CBAM brings order to the tornado of change. 
Journal of Staff Development, 19(4), 17-20. 

Marzano, R., Waters, T., & McNulty, B. (2005). School leadership that works. Aurora, CO: 
McREL. 

Newmann, F. M., Smith, B., Allensworth, E., & Bryk, A. S. (2001). School instructional 
program coherence: Benefits and challenges. Chicago, IL: Consortium on Chicago 
School Research. 

Rogers, E. M. (1995). Diffusion of innovations (4th ed.). New York: Simon & Schuster. 

Wenger, E., McDermott, R., & Snyder, W. M. (2002). Cultivating communities of practice: A 
guide to managing knowledge. Boston: Harvard Business School Press. 


This document was produced with funding from the U.S. Department of Education’s Office of 
Elementary and Secondary Education as part of ESETP grant S318A030029-05 to the North 
Carolina Department of Public Instruction for Looking at North Carolina’s Educational 
Technology (LANCET). The contents of the document do not necessarily reflect the views and 
policies of the US Department of Education or any other agency of the United States 
Government. 


Lessons Learned about Evaluation Capacity Building, November, 2007. 
SERVE Center at UNC-G 


16 



